Add nf-test sharding CI #1668

edmundmiller · 2024-10-01T19:01:57Z

ci: Copy over from methylseq
ci: Move Sarek specific settings to ci
ci: tap => xml

PR checklist

github-actions · 2024-10-01T19:34:28Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 245d355

+| ✅ 215 tests passed       |+
#| ❔  11 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
pipeline_todos - TODO string in base.config: Check the defaults for all processes

❔ Tests ignored:

files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: conf/modules.config
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_ci - actions_ci
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/sarek/sarek/.github/workflows/awstest.yml
template_strings - template_strings
modules_config - modules_config

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowSarek.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 3.5.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.step= mapping
nextflow_config - Config default value correct: params.split_fastq= 50000000
nextflow_config - Config default value correct: params.nucleotides_per_second= 200000
nextflow_config - Config default value correct: params.clip_r1= 0
nextflow_config - Config default value correct: params.clip_r2= 0
nextflow_config - Config default value correct: params.three_prime_clip_r1= 0
nextflow_config - Config default value correct: params.three_prime_clip_r2= 0
nextflow_config - Config default value correct: params.trim_nextseq= 0
nextflow_config - Config default value correct: params.length_required= 15
nextflow_config - Config default value correct: params.group_by_umi_strategy= Adjacency
nextflow_config - Config default value correct: params.aligner= bwa-mem
nextflow_config - Config default value correct: params.ascat_min_base_qual= 20
nextflow_config - Config default value correct: params.ascat_min_counts= 10
nextflow_config - Config default value correct: params.ascat_min_map_qual= 35
nextflow_config - Config default value correct: params.cf_coeff= 0.05
nextflow_config - Config default value correct: params.cf_contamination= 0
nextflow_config - Config default value correct: params.cf_minqual= 0
nextflow_config - Config default value correct: params.cf_mincov= 0
nextflow_config - Config default value correct: params.cf_ploidy= 2
nextflow_config - Config default value correct: params.sentieon_haplotyper_emit_mode= variant
nextflow_config - Config default value correct: params.sentieon_dnascope_emit_mode= variant
nextflow_config - Config default value correct: params.sentieon_dnascope_pcr_indel_model= CONSERVATIVE
nextflow_config - Config default value correct: params.dbnsfp_fields= rs_dbSNP,HGVSc_VEP,HGVSp_VEP,1000Gp3_EAS_AF,1000Gp3_AMR_AF,LRT_score,GERP++_RS,gnomAD_exomes_AF
nextflow_config - Config default value correct: params.vep_custom_args= --everything --filter_common --per_gene --total_length --offline --format vcf
nextflow_config - Config default value correct: params.vep_version= 111.0-0
nextflow_config - Config default value correct: params.vep_out_format= vcf
nextflow_config - Config default value correct: params.genome= GATK.GRCh38
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.vep_cache= s3://annotation-cache/vep_cache/
nextflow_config - Config default value correct: params.snpeff_cache= s3://annotation-cache/snpeff_cache/
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.test_data_base= https://raw.githubusercontent.com/nf-core/test-datasets/sarek3
nextflow_config - Config default value correct: params.seq_platform= ILLUMINA
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ncbench.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: cloudtest.yml
actions_schema_validation - Workflow validation passed: pytest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - UNZIP found in conf/base.config and Nextflow scripts.
base_config - FASTQC found in conf/base.config and Nextflow scripts.
base_config - FASTP found in conf/base.config and Nextflow scripts.
base_config - BWAMEM1_MEM found in conf/base.config and Nextflow scripts.
base_config - CNVKIT_BATCH found in conf/base.config and Nextflow scripts.
base_config - GATK4_MARKDUPLICATES found in conf/base.config and Nextflow scripts.
base_config - GATK4_APPLYBQSR found in conf/base.config and Nextflow scripts.
base_config - MOSDEPTH found in conf/base.config and Nextflow scripts.
base_config - STRELKA found in conf/base.config and Nextflow scripts.
base_config - SAMTOOLS_CONVERT found in conf/base.config and Nextflow scripts.
base_config - GATK4_MERGEVCFS found in conf/base.config and Nextflow scripts.
base_config - MULTIQC found in conf/base.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-10-15 11:45:53

To save runners

maxulysse · 2024-10-03T20:02:14Z

I guess now you'd have a couple of merge conflicts

.github/workflows/ci.yml

.github/nf-test-tags.yml

maxulysse · 2024-10-08T10:16:39Z

merged in #1678
Looking good

maxulysse · 2024-10-11T07:09:14Z

Synced with dev to include TEMPLATE merge

maxulysse · 2024-10-11T07:55:32Z

the last 2 failing tests are fixed in #1678

edmundmiller self-assigned this Oct 1, 2024

edmundmiller force-pushed the fresh-nf-test branch from 1d78ef8 to dccbb2e Compare October 1, 2024 19:27

edmundmiller added 3 commits October 1, 2024 14:30

ci: Copy over from methylseq

6839ad2

ci: Move Sarek specific settings to ci

6fdf037

ci: tap => xml

ac87c35

edmundmiller force-pushed the fresh-nf-test branch from dccbb2e to ac87c35 Compare October 1, 2024 19:30

edmundmiller added 3 commits October 1, 2024 14:43

build: Add triggers

50b9c88

ci: Add clean up

277c7fa

ci: Give jobs a name

fb36aa8

edmundmiller force-pushed the fresh-nf-test branch from 9731220 to 8236766 Compare October 2, 2024 13:52

edmundmiller marked this pull request as ready for review October 2, 2024 13:53

edmundmiller requested review from FriederikeHanssen and maxulysse as code owners October 2, 2024 13:53

edmundmiller added 3 commits October 2, 2024 09:04

ci: job-total => 5

ca0cfc3

ci: Follow dependencies

06f22b6

ci: Wondering if we can combine proccesses and functions

fcec8fa

To save runners

edmundmiller closed this Oct 2, 2024

edmundmiller force-pushed the fresh-nf-test branch from 8236766 to fcec8fa Compare October 2, 2024 14:04

edmundmiller added 2 commits October 2, 2024 09:35

test: Manually nump software versions txt

f44ffa3

ci: Run processes and functions seperately

5187cc1

edmundmiller reopened this Oct 2, 2024

ci(#1672): Remove latest-everything

ee7107f

Merge branch 'dev' into fresh-nf-test

0f4c1ed

maxulysse reviewed Oct 4, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

Update .github/workflows/ci.yml

89d1b07

maxulysse reviewed Oct 4, 2024

View reviewed changes

.github/nf-test-tags.yml Outdated Show resolved Hide resolved

maxulysse added 2 commits October 4, 2024 09:08

Update .github/nf-test-tags.yml

9edd171

no need for tags

bf661f4

maxulysse added 21 commits October 7, 2024 11:09

install pdiff

1d9a18e

properly install pdiff

b7417ec

update nftignore and snapshots

89507f8

Merge branch 'dev' into fresh-nf-test

574047a

update CHANGELOG

7b7b9c1

add tumor-normal-pair test

1bfe04a

update CHANGELOG

c9b2f0d

add more tests, update to nft-utils 0.0.3, add nft-bam 0.4.0

0c5114f

fix snapshots

977389f

ignore bam files from snapshots

d31edb8

ignore bai files from snapshots

1e82940

snapshot bam files

4713bad

update nftignore - no files from multiqc_plots

56791a2

update snapshots + snapshot bam files using nft-bam

2c9d25b

snapshot bam files

4db3c99

properly update nftignore - no files from multiqc_plots

5634713

snapshot cram files with nft-bam

6dd2fd6

update snapshots

61f19fd

update nftignore

ab16157

Merge branch 'nft-utils-0-0-3' into fresh-nf-test

d42727b

no need to exclude pipeline shards anymore

cba29dd

Merge branch 'dev' into fresh-nf-test

0dd1d41

maxulysse added 3 commits October 11, 2024 09:09

remove filter process for now

44d8f3b

bump minimum NXF_VER

4ba50b2

no status is 0

f470227

maxulysse mentioned this pull request Oct 11, 2024

Important! Template update for nf-core/tools v3.0.2 #1686

Merged

Merge branch 'dev' into fresh-nf-test

245d355

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nf-test sharding CI #1668

Add nf-test sharding CI #1668

edmundmiller commented Oct 1, 2024

github-actions bot commented Oct 1, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented Oct 3, 2024

maxulysse commented Oct 8, 2024

maxulysse commented Oct 11, 2024

maxulysse commented Oct 11, 2024

Add nf-test sharding CI #1668

Are you sure you want to change the base?

Add nf-test sharding CI #1668

Conversation

edmundmiller commented Oct 1, 2024

PR checklist

github-actions bot commented Oct 1, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

maxulysse commented Oct 3, 2024

maxulysse commented Oct 8, 2024

maxulysse commented Oct 11, 2024

maxulysse commented Oct 11, 2024

github-actions bot commented Oct 1, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️